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1.  Introduction 


As  robotic  assets  become  more  complex  and  autonomous,  there  are  challenges,  such  as  how  to 
keep  Soldiers  informed  about  robot  activity  and  intent,  as  well  as  maintain  overall  mission 
effectiveness,  without  increasing  workload.  Soldiers  need  to  be  able  to  quickly  gain  an 
understanding  of  a  robotic  asset’s  status  when  switching  from  one  task  to  another  task  (e.g., 
switching  between  reconnaissance  to  area  security).  Soldiers  also  need  to  have  some  predictive 
capability  about  a  robotic  asset’s  intended  courses  of  action,  so  that  corrective  actions  can  be 
taken  before  an  asset  gets  into  trouble.  In  this  study,  we  investigated  the  usefulness  of  operator 
aids,  which  overlay  information  about  unmanned  ground  vehicles’  (UGVs’)  perception  and 
planned  route  execution  onto  existing  user  interfaces,  as  a  way  to  address  these  challenges. 

The  goals  of  this  study  were  to  (1)  evaluate  the  extent  to  which  operator  perfonnance  and 
workload  is  affected  by  operator  aids  and  (2)  evaluate  possible  interaction  effects  of  operator  aid 
combinations.  For  the  purposes  of  this  study,  perfonnance  data  was  based  on  factors  established 
by  Fong  et  al.  (2004)  and  workload  was  evaluated  as  a  self-report  measure  using  the  NASA-Task 
Load  Index  (TLX)  (Hart  and  Staveland,  1988).  Additionally,  operator  preference  data  were 
collected  to  evaluate  which  of  the  operator  aids  were  preferred  by  the  participants  and  why. 

1.1  Background 

Jameson  (2001)  described  the  future  Anny  as  involving  extensive  use  of  mobile  sensing  systems, 
unmanned  platforms,  and  decision  aiding  systems.  He  also  remarked  that  information  fusion 
would  require  an  ability  to  create  and  maintain  real-time  situation  awareness  (SA)  from  all 
available  information.  Operator  aids  are  one  way  to  assist  robotic  operators  in  maintaining  SA  by 
quickly  displaying  mission  relevant  infonnation  without  requiring  too  much  attention  away  from 
primary  tasks.  Operator  aids  are  display  tools,  which  generate  information  from  data  supplied  by 
an  autonomous  robotic  asset  to  provide  an  increased  understanding  of  the  asset’s  situation  and 
actions.  The  intent  of  operator  aids  is  to  reduce  the  amount  of  cognitive  workload  by  supplying  an 
operator  with  infonnation  about  an  asset,  in  a  way  that  limits  the  need  for  additional  user  cognitive 
processing,  and  at  the  same  time  increasing  an  operator’s  understanding.  Wickens’  Multiple 
Resource  Theory  (1980,  1984)  suggests  that  as  workload  is  increased,  specifically  in  tasks  with 
overlapping  cognitive  resources  (i.e.,  vision-based  tasks,  such  as  reconnaissance  or  patrol  tasks), 
there  would  be  decreases  in  performance.  Therefore,  in  a  human-robot  team  reconnaissance 
task,  with  a  number  of  highly  visual  components,  it  is  expected  that  as  workload  increases, 
operator  performance  will  decline.  Mitchell  and  Brennan  (in  press)  reiterate  this  expectation 
within  their  Improved  Perfonnance  Research  Integration  Tool  (IMPRINT)  analysis,  predicting 
large  increases  in  workload  for  robotic  asset  operators  versus  nonoperators.  However,  we 
propose  that  the  appropriate  use  of  operator  aids  can  mitigate  some  of  this  increased  workload 
and  performance  decline  by  improving  the  information  presentation  by  means  of  operator  aids. 
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During  a  reconnaissance  mission  using  a  robotic  asset,  an  operator  must  supervise  the  asset, 
observe  the  mission  space,  locate  and  identify  potential  targets,  and  record  and  transmit  such 
information  to  decision  makers  up  the  chain  of  command.  These  tasks  are  in  addition  to 
maintaining  local  area  security  and  any  other  tasks  that  may  be  required  outside  of  those  related 
to  the  robotic  asset.  To  accomplish  all  of  these  tasks,  an  operator’s  cognitive  resources  are  sure  to 
be  taxed  (Mitchell,  2008).  Operator  aids  may  lessen  the  cognitive  burden.  Specifically,  in  this 
research,  operator  aids  will  be  focused  on  the  presentation  of  information  related  to  the  task  of 
supervising  the  robotic  asset.  By  providing  information  about  what  an  autonomous  robot 
calculates  as  its  best  route,  and  why  it  has  calculated  that  route,  operators  will  have  a  chance  to 
anticipate  trouble  spots,  provide  timely  input  when  needed,  and  better  understand  the 
autonomous  system’s  “intent.” 

A  robot  operator  who  obtains  target  infonnation  gained  from  a  robotic  asset’s  remote  sensors 
represents  the  first  step  in  providing  quality  information  to  decision  makers.  As  such,  it  is 
paramount  that  robot  operators  are  able  to  focus  attention  so  that  the  infonnation  they  provide  is 
as  accurate  as  possible.  Operator  aids  represent  tools  by  which  an  operator’s  workload  can  be 
reduced,  which  will  allow  more  resources  to  be  allocated  toward  other  mission  critical  tasks. 

1.2  Hypotheses 

Hypotheses  for  this  study  have  been  broken  up  into  (1)  hypotheses  related  to  perfonnance  and 
(2)  hypotheses  related  to  workload. 

1 .  Four  hypotheses  predicting  improved  perfonnance  for  experimental  condition  over  the 
control  condition  have  been  developed: 

•  Hi:  Participants  will  identify  simulated  targets  with  greater  accuracy. 

•  H2:  Participants  will  identify  unmanned  ground  vehicle  (UGV)  route  deviations  with 
greater  accuracy. 

•  H3:  Participants  will  engage  tele-operation  (tele-op)  control  less  often. 

•  H4:  Participants  will  complete  missions  in  less  time. 

2.  Two  hypotheses  were  formulated  in  relationship  to  the  workload: 

•  H5:  Participants  will  experience  lower  levels  of  perceived  workload  in  the  experimental 
conditions  than  in  the  control  condition. 

•  He:  Participants’  feedback  will  indicate  a  preference  for  operating  a  robotic  asset  with 
operator  aids  as  opposed  to  having  no  operator  aids  available. 
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2.  Methodology 


2.1  Instrumentation  and  Facilities 

This  study  utilized  ModSim  simulation  software,  developed  by  General  Dynamics  Robotics 
System  (GDRS).  This  simulation  was  created  as  a  visualization  platform  that  serves  as  a  basis  for 
realistic  simulated  experimentation,  validation,  and  refinement  of  hardware-in-the-loop  dynamic 
planners.  This  simulation  was  used  in  conjunction  with  the  U.S.  Army  Tank  Automotive 
Research,  Development,  and  Engineering  Center’s  (TARDEC’s)  Warfighter  Machine  Interface 
(WMI),  developed  by  DCS  Corp.  The  WMI  is  an  interface  system  designed  to  accommodate  a 
number  of  different  technologies  through  the  use  of  various  tabs  for  access.  Examples  of  this 
interface  can  be  seen  in  figures  1  and  2.  Participants  monitored  the  forward-facing  camera  of  the 
simulated  UGV  and  entered  task  information  using  a  touch  screen  monitor.  All  of  the  software 
was  run  on  standard  PCs,  using  two  standard  17  inch  color  monitors. 


Figure  1.  An  example  of  the  WMI  3-D  interface  showing 
both  the  short-term  (green  line)  and  long-term 
(blue  line)  operator  aids. 
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Figure  2.  An  example  of  the  WMI  2-D  interface  showing  the  obstacle  map 
overlay  on  a  top-down  map.  The  edges  of  obstacle  are 
represented  with  red  dots  and  the  areas  shaded  with  pink  are 
unknown  due  to  an  obstructed  view  from  the  UGV. 

The  WMI  display  has  two  screens  showing  different  views  of  the  environment.  The  forward¬ 
facing  video  display  is  referred  to  as  the  three-dimensional  (3-D)  display.  It  is  called  the  3-D 
display  because  this  display  shows  a  live  feed  of  the  environment  directly  in  front  of  the  UGV 
and  allows  for  a  ground  level  view  of  the  mission  space.  An  example  of  this  can  be  seen  in 
figure  1 .  An  overhead  map  view  of  the  environment,  referred  to  as  the  two-dimensional  (2-D) 
display,  shows  an  overhead  “satellite”  view  of  the  environment,  representative  of  a  2-D  map. 

The  2-D  view  also  gives  users  a  reference  about  the  UGV’s  location  within  the  mission  space. 
An  example  of  the  2-D  portion  of  the  WMI  display  can  be  seen  in  figure  2.  The  2-D  and  3-D 
screens  were  continuously  displayed  on  the  two  17-inch  monitors  simultaneously,  so  that 
participants  could  attend  to  either  screen  at  any  time  during  the  experiment. 

Autonomous  navigation  for  the  simulated  UGV  was  handled  using  GDRS’  Autonomous 
Navigation  System  (ANS)  software.  This  software  was  identical  to  the  software  used  for  a  real 
world  version  of  the  UGV  and  allowed  the  vehicle  to  navigate  an  environment  via  waypoint  plan 
while  avoiding  environmental  obstacles  (GDRS,  2013). 

2.1.1  Courses  or  Facilities 

This  study  took  place  at  indoor,  office-like  locations  at  the  U.S.  Army  Research  Laboratory 
(ARL)  at  Aberdeen  Proving  Ground,  MD  and  DCS  Corp.  laboratories  in  Alexandria,  VA. 
Participants  completed  the  study  while  in  a  seated  position  at  a  computer  station  approximately 
18  inches  directly  in  front  of  the  two  display  screens. 
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2.2  Materials  and  Tests 

Participants  completed  a  number  of  assessments  designed  to  assess  their  color  vision  acuity, 
spatial  visualization  ability,  and  spatial  orientation  ability,  as  well  as  a  demographic 
questionnaire  to  characterize  the  participant  population. 

To  assess  color  vision  acuity,  the  Ishihara  (1917)  color  vision  test  was  used  (appendix  A)  as  a 
screening  procedure.  The  color  vision  test  was  used  to  ensure  that  participants  have  no  unknown 
color  vision  deficiencies,  as  color  is  an  important  component  of  the  WMI  and  operator  aids. 

Spatial  visualization  and  spatial  orientation  were  assessed  using  the  portions  of  the 
Guilford-Zimmerman  (1956)  attribute  survey  related  to  these  two  areas  (appendices  B  and  C). 
Each  of  these  assessments  has  been  used  previously  in  studies  investigating  human-robot 
interaction  (HRI)  (Fincannon  et  ah,  2008),  and  have  shown  positive  corollary  effects  with  robot 
operator  performance.  These  effects  indicated  that  individuals  displaying  higher  levels  of  spatial 
ability  generally  show  higher  levels  of  performance  when  engaging  in  HRI  supervision  tasks. 

The  spatial  ability  scores  were  used  as  covariates  in  the  data  analysis. 

Participants’  perceived  workload  was  evaluated  after  the  completion  of  each  mission  scenario 
using  the  NASA-TLX  questionnaire  (Hart  and  Staveland,  1988,  see  appendix  D).  The  NASA- 
TLX  is  a  self-report  questionnaire  of  perceived  demands  in  six  areas:  mental,  physical,  temporal, 
effort  (mental  and  physical),  frustration,  and  performance.  Participants  were  asked  to  rate  their 
perceived  workload  level  in  these  six  areas  on  100-point  scales,  indicated  with  “high”  and  “low” 
ends.  The  ratings  were  used  to  quantify  the  perceived  changes  in  workload  for  the  various 
operator  aids  used. 

Finally,  participants’  operator  aid  configuration  preference  and  display  use  was  recorded  after 
the  seven  trials  by  simply  asking  each  participant  what  their  favorite  and  least  favorite  aid 
configurations  were  and  how  much  they  used  the  2-D  and  3-D  screens. 

2.3  Experimental  Design 

Participants  in  this  study  completed  a  series  of  reconnaissance  missions  in  which  they  identified 
targets  from  a  simulated  autonomous  UGV  platform.  Mission  navigation  of  the  UGV  was  fully 
autonomous,  however  the  UGV  platform  interface  included  a  variety  of  operator  aids  intended  to 
inform  the  operator  about  robot  intent  and  robot  obstacle  identification.  Participants  only  took 
tele-op  (remote)  control  of  the  UGV  when  it  was  encountering  an  obstacle  (discussed  further  in 
the  procedure  section  of  this  report). 

This  study  employed  a  3  (Operator  Aid  Condition)  x  2  (Obstacle  Map  Condition)  +  1  (Control 
Group  of  no  operator  aids)  within- subjects  design.  This  design  was  required  due  to  technology 
constraints,  which  limited  the  application  of  the  Obstacle  Map  aid,  thus  not  allowing  it  to  be 
evaluated  separately  from  the  other  conditions.  For  analysis  purposes,  each  experimental  cell  was 


5 


compared  with  the  control  cell,  using  a  one-way  analysis  of  variance  (ANOVA)  to  test  for 
significance  among  the  seven  conditions.  This  analysis  allowed  researchers  to  understand  if 
operator  aids  presented  as  a  single  aid  or  multiple  aids  contributed  to  improved  performance  over 
a  condition  involving  no  aids. 

2.3.1  Independent  Variables 

Operator  Aids.  The  independent  variable  levels  for  Operator  Aid  Condition  were  (1)  short-term 
planner  (STP)  only,  (2)  long-tenn  planner  (LTP)  only,  and  (3)  combined  short-term  and  long¬ 
term  planners  (COMBO).  Both  the  STP  and  LTP  were  generated  using  data  from  the  ANS  about 
the  path  that  the  automation  attempted  to  follow.  The  STP  operator  aid  was  displayed  as  a 
translucent  green  line  overlaid  on  both  forward-facing  video  stream  (3-D)  and  overhead  map 
(2-D)  views  of  the  WMI.  The  line  for  the  STP  operator  aid  displayed  the  intended  route  of  the 
UGV  projected  for  the  next  several  seconds.  Similarly,  the  LTP  operator  aid  was  displayed  as  a 
translucent  blue  line  overlaid  on  both  forward-facing  video  stream  and  overhead  map  views  of 
the  WMI  interface.  However,  the  LTP  operator  aid  displayed  infonnation  about  the  intended 
route  of  the  UGV  projected  for  the  next  several  minutes  or  more.  The  combination  of  STP  and 
LTP  operator  aids  simply  displayed  both  translucent  green  and  blue  lines  at  the  same  time.  See 
figures  1  and  2  for  an  example  of  the  displays.  The  two  lines  representing  the  STP  and  LTP  may 
or  may  not  have  overlapped,  depending  on  the  situation.  The  actual  length  of  the  two  lines  was 
dependent  upon  vehicle  speed  and  varied  accordingly. 

Obstacle  Map.  The  independent  variable  levels  for  Obstacle  Map  Condition  were  (1)  with  the 
obstacle  map  overlays  and  (2)  without  the  obstacle  map  overlays.  Obstacle  Maps  were  generated 
from  laser  radar  (LADAR)  data  acquired  by  the  ANS,  which  were  then  transformed  to  display 
the  outlines  of  physical  obstacles  in  the  environment.  Scenarios  with  the  obstacle  map  included 
translucent  overlays  on  both  forward-facing  video  stream  and  overhead  map  views  showing 
obstacles  that  the  UGV  had  detected.  On  the  video  feed  view,  the  obstacle  map  was  represented 
via  red  (nonpassable  obstacle)  and  green  (passable  obstacle)  squares  at  ground  level  (see 
figure  1).  For  the  overhead  view,  obstacle  maps  were  represented  using  red  pixels  to  show 
nonpassable  obstacle  borders  and  pink  translucent  shading  to  indicate  unknown  areas  of  the  map 
(see  figure  2).  Overhead  maps  were  preset  at  a  fixed  zoom  scale  to  ensure  that  all  participants 
received  the  same  view. 

2.3.2  Dependent  Variables 

Dependent  variables  for  this  study  included  (1)  number  of  accurate  target  identifications,  (2) 
number  of  unintentional  route  deviations  missed,  (3)  number  of  times  engaging  tele-operation 
mode,  and  (4)  total  mission  time.  These  are  all  variants  of  well-established  metrics  of 
performance  that  are  frequently  used  in  HRI  research  (Fong  et  ah,  2004).  “Number  of  accurate 
target  identifications”  is  defined  as  the  number  of  correct  target  locations  identified  on  the 
overhead  map  during  the  performance  mission.  Target  locations  were  identified  by  operators 
touching  the  location  on  the  interface.  “Number  of  unintentional  route  deviations  missed”  is 
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defined  as  the  total  number  of  times  during  the  perfonnance  mission  that  participants  fail  to 
identify,  or  incorrectly  identify,  why  a  deviation  from  the  preplanned  mission  route  has  occurred. 
Responses  were  recorded  from  verbal  declarations.  “Number  of  times  engaging  tele-operation 
mode”  is  defined  as  the  total  number  of  times  the  participant  engages  the  tele-operation  mode  of 
UGV  control  during  a  perfonnance  mission.  This  metric  was  available  from  the  data  logs.  Again 
using  information  available  from  data  logs,  “Total  mission  time”  is  defined  as  the  total  amount  of 
time,  in  seconds,  it  takes  to  complete  a  perfonnance  mission  beginning  with  the  moment  that  an 
automated  route  plan  has  begun  executing  to  the  completion  of  that  plan,  including  all  pauses 
and  occurrences  of  tele-operation  control.  All  of  this  infonnation  was  gathered  from  either 
written  notes  to  verbal  responses  or  as  .log  files  created  for  each  mission.  The  data  from  these 
files  were  reduced  and  input  into  statistical  software  to  investigate  the  appropriate  measures. 

The  NASA-TLX  was  used  to  measure  ratings  of  perceived  workload,  resulting  in  a  composite 
score  derived  from  six  areas  of  workload  (mental,  physical,  temporal,  effort,  frustration,  and 
performance).  In  addition,  data  were  collected  about  participants’  spatial  abilities  (via  the 
Guilford-Zimmerman  Spatial  Aptitude  survey)  to  be  used  as  covariates  in  data  analysis. 

Finally,  participants’  preferences  were  recorded  for  which  operator  aids  were  preferred  or 
disliked,  as  well  as  utilization  data  referring  to  which  of  the  two  display  screens  (2-D  or  3-D) 
was  used  more.  These  data  were  gathered  by  simply  asking  each  participant  at  the  end  of  the 
experiment  “which  condition  was  their  favorite?”,  “which  condition  was  their  least  favorite?”, 
and  to  “estimate  the  percentage  of  time  spent  looking  at  the  2-D  screen  and  3-D  screen.” 

2.4  Procedure 

Participants  completed  seven  counterbalanced  experimental  scenarios  designed  to  explore  all  of 
the  possible  independent  variable  combinations  in  the  3  (Operator  Aid  Combination)  x  2 
(Obstacle  Map  Availability)  design,  plus  one  control  condition  that  had  no  operator  aids 
displayed.  Once  an  informed  consent  was  completed,  participants  were  given  the  color  vision 
test.  If  participants  exhibited  any  color  vision  impairment,  they  would  be  removed  from  the  study 
(none  did).  Next,  each  participant  was  given  a  demographic  questionnaire  to  fill  out.  Finally, 
participants  were  given  the  two  assessments  of  their  spatial  ability,  one  for  spatial  visualization 
ability  and  one  for  spatial  orientation  ability. 

The  participants  received  a  short  tutorial  on  the  use  of  the  simulation  interface  and  instructions 
on  how  to  complete  the  reconnaissance  task.  The  simulation  was  a  representation  of  a  UGV 
traversing  a  predetermined  route  while  participants  monitored  the  displays,  searching  for  targets 
(represented  as  “smiley  faces”)  along  the  route  traveled.  Participants  were  given  an  opportunity 
to  observe  the  2-D  and  3-D  displays  and  use  the  simulation  system  in  a  practice  scenario  to 
increase  familiarity  with  the  system.  This  gave  participants  an  opportunity  to  review  the  various 
target  and  obstacle  items  that  they  would  encounter  during  the  experimental  scenarios. 
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After  familiarization  and  training,  participants  were  asked  to  act  as  operator  for  a  UGV  on  a 
reconnaissance  mission.  Each  reconnaissance  scenario  consisted  of  one  UGV  following  a 
predetennined  route  through  the  urban  terrain.  Along  the  route,  participants  had  the  opportunity 
to  identify  up  to  six  separate  targets,  still  represented  as  “smiley  faces.”  In  addition  to  scanning 
for  targets,  participants  monitored  the  UGV  to  ensure  that  it  remained  on  its  intended  path.  Each 
mission  included  a  set  (three  total)  of  potential  obstacles  that  could  cause  the  UGV’s  ANS  to 
deviate  from  its  planned  path.  These  obstacles  were  positioned  randomly  throughout  the  route 
and  consisted  of  either  a  patch  of  tall  grass,  a  person  crossing  the  road,  or  a  Jersey  barrier 
concrete  wall.  These  obstacle  types  were  chosen  because  they  all  cause  similar  issues  for  current 
ANS  but  have  different  courses  of  action  associated  with  an  encounter.  Participants  were  asked 
to  choose  an  appropriate  course  of  action  (COA)  for  each  obstacle  encountered.  The  three  COAs 
were  (1)  continue  as  planned,  (2)  wait  for  obstacle  to  pass  and  then  continue,  or  (3)  reroute  the 
UGV.  When  tall  grass  was  encountered  and  caused  a  route  deviation  from  the  plan,  the 
appropriate  COA  was  to  instruct  the  UGV  to  continue  on  the  planned  route  without  stopping 
(“continue”).  For  moving  person  obstacles,  the  appropriate  COA  was  to  instruct  the  UGV  to  wait 
for  the  obstacle  to  clear  and  then  continue  on  the  planned  route  (“wait  and  continue”).  Finally,  if 
an  unexpected  Jersey  wall  appeared  as  an  obstacle,  a  new  route  was  needed  to  be  planned 
(“reroute”).  COAs  were  executed  by  speaking  aloud  the  obstacle  type  being  encountered  and 
then  operating  the  UGV  with  the  appropriate  maneuver  for  the  given  obstacle.  At  the  end  of  each 
of  the  seven  scenarios,  NASA-TLX  data  was  collected  and  at  the  completion  of  all  the  scenarios, 
user  preference  and  use  data  were  collected  via  interview. 


3.  Performance  Data  Results  and  Analysis 


For  this  study,  Noncommissioned  Officers  (NCOs)  and  civilians  from  the  local  ARE  population 
were  recruited  as  participants.  A  total  of  20  individuals  participated.  Participants  included  15 
males  and  5  females  with  an  average  age  of  32.2  years  (standard  deviation  =  6.895).  Of  the  20 
total  participants,  four  were  active  duty  NCOs  in  the  U.S.  Anny  and  one  was  a  cadet  at  the  U.S. 
Military  Academy. 

To  investigate  the  performance-related  hypotheses,  comparisons  of  the  experimental  conditions 
against  the  control  were  completed  for  each  of  the  dependent  variables,  analyzed  with  one-way 
ANOVAs  using  Statistical  Package  for  the  Social  Sciences  (SPSS)  version  19.0.  Results  of  the 
individual  ANOVAs  are  presented  in  the  following  sections. 
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3.1  Target  Identification 


A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  Operator  Aid  Condition  on 
participants’  ability  to  accurately  identify  targets  in  the  experimental  environment.  Results  of  the 
ANOVA  showed  no  significant  differences  among  the  experimental  conditions  ( F(6,90 )  =  1.652, 
p  =  0. 142).  To  ensure  that  other  factors  were  not  influencing  the  analysis,  covariate  data  was 
used  in  a  followup  ANCOVA  analysis.  Covariates  were  derived  from  the  two  spatial  ability 
evaluations:  spatial  orientation  and  spatial  visualization.  The  covariates,  representing  an 
individual  participant’s  spatial  ability,  were  derived  from  the  total  number  of  correct  answers 
given  per  questionnaire.  The  covariate  analysis  was  used  to  control  specifically  for  ability, 
because  it  is  known  that  spatial  ability  is  positively  correlated  with  robotic  operator  performance 
(Fincannon  et  ah,  2008).  Therefore,  the  results  for  differences  in  performance  should  be 
attributed  to  the  experimental  conditions  (i.e.,  operator  aids)  and  not  just  because  the  participants 
might  have  differed  in  innate  ability.  The  same  approach  for  covariates  was  used  for  each  of  the 
performance  measure  analysis  (sections  3. 1-3.4).  Results  of  the  ANCOVA  showed  no 
significant  influence  from  any  of  the  covariates  and  no  significant  differences  in  the  results 
(F(6, 78)  =  0.607,  p  =  0.724)  and  therefore  unadjusted  data  were  used  for  analysis.  The  data  from 
this  analysis  have  been  summarized  in  table  1 . 

Table  1.  Means  and  standard  deviations  for  target  identification  data  in  the  seven  conditions 
(measured  in  correct  identifications  per  mission,  i.e.,  larger  score  is  better, 
maximum  =  6). 


Condition 

Mean 

Standard  Deviation 

Cl:  Short-term  and  obstacle  map 

4.75 

1.24 

C2:  Short-term  only 

5.00 

0.97 

C3:  Long-term  and  obstacle  map 

4.38 

1.54 

C4:  Long-term  only 

4.63 

1.46 

C5:  Short-term,  long-term,  and  obstacle  map 

4.69 

1.62 

C6:  Short-term  and  long-term 

4.31 

1.49 

Cl:  No  operator  aids 

3.88 

1.54 

3.2  Route  Deviations 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  operator  aid  condition  on 
participants’  ability  to  accurately  identify  UGV  unintentional  route  deviations  in  the 
experimental  environment.  Results  of  the  ANOVA  showed  no  significant  differences  in  the 
experimental  conditions  (F(6,114)  =  1.029,  p  =  0.410).  To  ensure  that  other  factors  were  not 
influencing  the  analysis,  covariate  data  was  used  in  a  follow  up  ANCOVA  analysis.  The  same 
approach  for  covariates  was  used  as  explained  in  section  3.1.  Results  of  the  ANCOVA  showed 
no  significant  influence  from  any  of  the  covariates  and  no  significant  differences  in  the  results 
(. F(6,102 )  =  0.611,  p  =  0.669).  The  data  from  this  analysis  have  been  summarized  in  table  2. 
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Table  2.  Means  and  standard  deviations  for  unintentional  route  deviation  data  in  the  seven  conditions 
(measured  in  number  of  missed  route  deviations  per  mission,  i.e.,  smaller  score  is  better, 
minimum  =  0). 


Condition 

Mean 

Standard  Deviation 

Cl:  Short-term  and  obstacle  map 

0.40 

0.503 

C2:  Short-term  only 

0.40 

0.503 

C3 :  Long-term  and  obstacle  map 

0.35 

0.587 

C4:  Long-term  only 

0.30 

0.470 

C5:  Short-term,  long-term,  and  obstacle  map 

0.30 

0.470 

C6:  Short-term  and  long-term 

0.25 

0.444 

C7:  No  operator  aids 

0.60 

0.598 

3.3  Tele-Operation  Occurrences 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  Operator  Aid  Condition  on 
participants’  use  of  the  tele-operation  control  mode  in  the  experimental  environment.  Results  of 
the  ANOVA  showed  no  significant  differences  in  the  experimental  conditions  ( F(6,114 )  =  0.533, 
p  =  0.782).  To  ensure  that  other  factors  were  not  influencing  the  analysis,  covariate  data,  as 
previously  described,  was  used  in  a  follow  up  ANCOVA  analysis.  Results  of  the  ANCOVA 
showed  no  significant  influence  from  any  of  the  covariates  and  no  significant  differences  in  the 
results  ( F(6,102 )  =  0.564,  p  =  0.758).  The  data  from  this  analysis  have  been  summarized  in 
table  3.  Based  on  scenario  design,  we  expected  that  each  individual  would  have  a  minimum  of 
three  tele-operation  occurrences  per  mission. 

Table  3.  Means  and  standard  deviations  for  tele-operation  occurrences  data  in  the  seven  conditions  (measured 
in  number  of  times  tele-operation  mode  was  engaged  per  mission,  i.e.,  smaller  score  is  better). 


Condition 

Mean 

Standard  Deviation 

Cl:  Short-term  and  obstacle  map 

3.50 

1.00 

C2:  Short-term  only 

4.00 

0.86 

C3 :  Long-term  and  obstacle  map 

3.75 

1.02 

C4:  Long-term  only 

3.80 

0.77 

C5:  Short-term,  long-term,  and  obstacle  map 

3.80 

0.89 

C6:  Short-term  and  long-term 

3.70 

0.66 

C7:  No  operator  aids 

3.75 

1.21 

3.4  Total  Mission  Time 

Finally,  a  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition 
on  overall  mission  time.  Results  of  the  ANOVA  showed  no  significant  differences  in  the 
experimental  conditions  ( F(6,102 )  =  0.714,  p  =  0.639).  To  ensure  that  other  factors  were  not 
influencing  the  analysis,  covariate  data  was  used  in  a  follow  up  ANCOVA  analysis.  Covariates 
were  derived  from  the  spatial  ability  evaluations  mentioned  earlier  and  scored  as  described 
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above.  Results  of  the  ANCOVA  showed  no  significant  influence  from  any  of  the  covariates  and 
no  significant  differences  in  the  results  ( F(6,90 )  =  0.725,  p  =  0.630).  The  data  from  this  analysis 
have  been  summarized  in  table  4. 

Table  4.  Means  and  standard  deviations  for  total  mission  time  data  in  the  seven  conditions  (measured  in  total 

seconds  from  beginning  of  automated  plan  execution  to  completions  of  plan  per  mission,  i.e.,  less  time  is 
better). 


Condition 

Mean 

Standard  Deviation 

Cl:  Short-term  and  obstacle  map 

406.98 

34.50 

C2:  Short-term  only 

401.28 

50.57 

C3 :  Long-term  and  obstacle  map 

412.11 

39.51 

C4:  Long-term  only 

398.26 

37.32 

C5:  Short-term,  long-term,  and  obstacle  map 

406.21 

30.93 

C6:  Short-term  and  long-term 

419.38 

35.08 

C7:  No  operator  aids 

404.96 

38.53 

4.  Workload  Data  Analysis 


To  investigate  the  workload  hypothesis,  comparisons  of  the  experimental  conditions  against  the 
control  were  completed.  NASA-TLX  composite  scores,  calculated  by  summing  the  six  subscale 
scores  and  dividing  by  6  (highest  possible  score  is  100),  were  calculated  and  then  analyzed  with 
a  one-way  ANOVA  using  SPSS  19.0  to  evaluate  the  effects  of  the  Operator  Aid  Condition  on 
self-reported  workload  as  measured  by  the  NASA-TLX.  Results  of  the  ANOVA  showed  a 
significant  effect  existed  (F (6,1 14)  =  97.188,  p  <  0.001),  among  the  self-reported  workload 
measures  for  the  different  Operator  Aid  Conditions. 

Post  hoc  analysis,  using  paired  sample  t-test  (t(19)  =  -2.071,  p  =  0.026,  one-tailed)  and  Tukey 
HSD  correction  for  Type  I  error  showed  that  the  condition  with  the  STP  and  LTP  operator  aids 
only  (Condition  6:  short-term  planner,  long-term  planner,  no  obstacle  map)  yielded  significantly 
lower  composite  workload  scores  than  the  control  condition  without  any  operator  aids  (Condition 
7).  A  marginal  finding  involved  the  condition  that  included  both  long-term  and  short-term 
planners  and  the  obstacle  map  (Condition  5).  T-test  analysis  (t(  19)  =  -1.676,  p  =  0.055,  one- 
tailed)  showed  this  condition  to  yield  lower  amounts  of  self-reported  workload  than  the  control 
condition  (Condition  7),  though  these  results  were  not  significant  even  with  the  benefit  of  a  one- 
tailed  test.  No  other  results  showed  significance.  These  results  are  depicted  in  figure  3. 
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Condition 


Figure  3.  Means  plot  for  NASA-TLX  scores  by  experimental  condition  (1=  Short-Term  Planner  and  Obstacle 
Map;  2=  Short-Term  Planner  only;  3=  Long-Term  Planner  and  Obstacle  Map;  4=  Long-Term  Planner 
only;  5=  Short-Term  Planner,  Long-Term  Planner,  and  Obstacle  Map;  6=  Short-Term  Planner  and 
Long-Term  Planner;  7  =  Control  -  no  operator  aids). 


A  significant  workload  composite  score  result  prompted  analyses  of  the  individual  workload 
subscales  (as  suggested  by  Hill  et  al.  [1992])  to  determine  what  specific  subscales  of  workload 
were  affected  by  the  inclusion  of  the  aids.  The  analyses  revealed  that  only  two  of  the  NASA- 
TLX  subscales  showed  significant  results:  mental  workload  and  temporal  workload.  All  subscale 
results  are  shown  for  completeness,  including  no  significant  results.  For  each  subscale,  a 
maximum  rating  is  100.  Table  5  shows  the  scores  for  each  subscale  and  the  composite  score. 
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Table  5.  Mean  scores  for  the  composite  NASA-TLX  and  subscales  by  experimental  condition. 


Composite 

Mental 

Physical 

Temporal 

Performance 

Effort 

Frustration 

Condition  1 

31.47 

44.50 

12.00 

30.25 

30.30 

48.75 

23.05 

Condition  2 

30.00 

40.80 

9.90 

30.60 

30.60 

45.50 

22.55 

Condition  3 

32.80 

46.65 

11.05 

34.40 

32.70 

46.15 

25.85 

Condition  4 

30.60 

42.30 

10.05 

28.60 

34.10 

44.50 

24.05 

Condition  5 

29.78 

38.65 

13.05 

28.60 

38.85 

42.85 

22.70 

Condition  6 

29.58 

37.70 

11.75 

32.60 

31.75 

43.80 

19.90 

Condition  7 

33.31 

47.00 

13.70 

33.95 

34.95 

45.40 

24.85 

Notes:  Maximum  =  100.  1=  Short-Term  Planner  and  Obstacle  Map;  2=  Short-Term  Planner  only;  3=  Long-Term  Planner  and 
Obstacle  Map;  4=  Long-Term  Planner  only;  5=  Short-Term  Planner,  Long-Term  Planner,  and  Obstacle  Map;  6=  Short- 
Term  Planner  and  Long-Term  Planner;  7=  Control  -  no  operator  aids. 


4.1  Mental  Workload 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition  on  self- 
reported  mental  workload  as  reported  using  the  NASA-TLX.  It  showed  significant  differences 
{¥(6,114)  =  82.955,  p  <  0.001)  among  experimental  conditions  (see  figure  4). 


Figure  4.  Means  plot  for  NASA-TLX  Mental  Workload  scores  by  experimental 

condition  (1=  Short-Term  Planner  and  Obstacle  Map;  2=  Short-Term  Planner 
only;  3=  Long-Term  Planner  and  Obstacle  Map;  4=  Long-Term  Planner  only; 
5=  Short-Term  Planner,  Long-Term  Planner,  and  Obstacle  Map;  6=  Short- 
Term  Planner  and  Long-Term  Planner;  7=  Control  -  no  operator  aids). 
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Post  hoc  analysis,  using  paired  sample  t-test  (t(19)  =  -1.953,  p  =  0.033,  one-tailed)  and  Tukey 
HSD  correction  for  Type  I  error  showed  that  the  condition  in  which  all  three  of  the  operator  aids 
(short-term  planner,  long-tenn  planner,  and  obstacle  map;  Condition  5)  were  present  and  yielded 
significantly  lower  mental  workload  than  the  control  condition  without  any  operator  aids 
(Condition  7).  The  only  other  experimental  condition  to  show  significant  results  was  the 
condition  that  included  both  long-tenn  and  short-tenn  planners  but  without  the  aid  of  the 
obstacle  map  (Condition  6).  T-test  analysis  (t(19)  =  -2.374,  p  =  0.014,  one-tailed)  showed  this 
condition  to  yield  significantly  lower  amounts  of  self-reported  mental  workload  than  the  control 
condition  (Condition  7).  No  other  experimental  conditions  compared  to  the  control  condition 
yielded  significant  results  in  tenns  of  mental  workload. 

4.2  Physical  Workload 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition  on  self- 
reported  physical  workload  as  reported  using  the  NASA-TLX.  It  showed  no  significant 
differences  ( F(6,114 )  =  0.241,  p  =  0.962)  among  experimental  conditions. 

4.3  Temporal  Workload 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition  on  self- 
reported  temporal  workload.  Results  of  the  ANOVA  showed  a  significant  effect  existed 
{¥(6,114)  =  7.993,  p  =  0.012)  among  the  self-reported  workload  measures  for  temporal  demand. 

Post-hoc  analysis,  using  paired  sample  t-test  (?(19)  =  -2.374,  p  =  0.014,  one-tailed)  and  Tukey 
HSD  correction  for  Type  I  error  showed  that,  again,  the  condition  in  which  all  three  of  the 
operator  aids  (short-term  planner,  long-tenn  planner,  and  obstacle  map;  Condition  5)  were 
present  yielded  significantly  lower  temporal  workload  than  the  control  condition  without  any 
operator  aids  (Condition  7).  One  other  experimental  condition  showed  significant  results  for 
temporal  workload,  the  condition  that  included  only  the  long-term  planner  aid  (Condition  4).  T- 
test  analysis  (f(19)  =  -1.752,  p  =  0.050,  one-tailed)  showed  this  condition  to  yield  significantly 
lower  amounts  of  self-reported  temporal  workload  than  the  control  condition  (Condition  7).  No 
other  experimental  conditions  compared  to  the  control  condition  yielded  significant  results  in 
terms  of  temporal  workload  (see  figure  5). 
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Figure  5.  Means  plot  for  NASA-TLX  Temporal  Workload  scores  by  experimental 

condition  (1=  Short-Term  Planner  and  Obstacle  Map;  2=  Short-Term  Planner 
only;  3=  Long-Term  Planner  and  Obstacle  Map;  4=  Long-Term  Planner  only; 

5=  Short-Term  Planner,  Long-Term  Planner,  and  Obstacle  Map;  6=  Short-Term 
Planner  and  Long-Term  Planner;  7=  Control  -  no  operator  aids). 

4.4  Performance  Workload 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition  on  self- 
reported  performance  as  reported  using  the  NASA-TLX.  It  showed  no  significant  differences 
(F (6,114  =  0.146,  p  =  0.990)  among  experimental  conditions. 

4.5  Effort  Workload 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition  on  self- 
reported  effort  as  reported  using  the  NASA-TLX.  It  showed  no  significant  differences  (F (6,1 14) 
=  0.100,  p  =  0.996)  among  experimental  conditions. 

4.6  Frustration  Workload 

A  one-way  ANOVA  was  conducted  to  evaluate  the  effect  of  the  Operator  Aid  Condition  on  self- 
reported  frustration  as  reported  using  the  NASA-TLX.  It  showed  no  significant  differences 
(F (6,114)  =  0.133,  p  =  0.992)  among  experimental  conditions. 
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4.7  User  Preferences 


4.7.1  Operator  Aid  Condition 

Participants  preferences  were  evaluated  for  which  of  the  conditions  were  their  favorite  and  least 
favorite  to  use,  based  on  the  poststudy  interview  questions  described  in  section  2.3.  Although  the 
choices  for  favorite  condition  were  more  varied,  the  majority  of  the  choices  (18  of  20 
participants)  included  experimental  conditions  that  involved  at  least  two  of  the  three  operator 
aids.  Additionally,  the  majority  of  subjects  (15  of  20)  listed  the  control  condition,  which  had  no 
operator  aids  as  their  least  favorite  condition  (see  table  6). 

Table  6.  Participant’s  favorite  and  least  favorite  operator  aid  configurations. 


Conditions 

Favorite  Condition 

Least  Favorite  Condition 

Short-  and  long-term  planners  with  obstacle  map 

4 

1 

Short-term  planner  and  obstacle  map 

7 

2 

Long-term  planner  and  obstacle  map 

3 

0 

Short-term  and  long-term  planners 

4 

0 

Short-term  planner  only 

0 

1 

Long-term  planner  only 

1 

1 

No  operator  aids  (control) 

1 

15 

4.7.2  2-D  (Overhead  View)  vs.  3-D  (Forward-Facing  View)  Use 

As  part  of  the  post  study  interview  questions  described  in  section  2.3,  participants  reported  using 
both  the  3-D  (front-facing  view)  and  2-D  (overhead  view)  displays  to  complete  the  task  but  with 
varying  frequency.  Results  show  that  3-D  display  was  reported  as  being  used  more  than  70% 
(mean  =  71.1%,  standard  deviation  =  13.5)  of  the  time  during  missions,  whereas  the  2-D  display 
was  used  less  than  30%  (mean  =  28.9%,  standard  deviation  =  13.5)  of  the  time.  This  information 
gives  researchers  a  baseline  to  understand  how  operators  are  using  the  2-D  and  3-D  views  to 
complete  the  mission  tasks.  Additionally,  this  information  could  be  used  to  determine  how  best 
to  present  various  operator  aid  types  and  how  they  are  most  useful. 


5.  Discussion 


5.1  Performance 

Results  of  the  performance  data  analysis  failed  to  support  any  of  the  four  hypotheses  associated 
with  the  perfonnance  measurements.  This  failure  to  observe  any  significant  differences  may  be 
due  to  several  factors.  A  relatively  simple  task  could  have  affected  results.  Additionally, 
participants  in  this  study  received  a  minimum  amount  of  training  and  were  exposed  to  the 
operator  aids  for  only  a  short  time.  More  training  and  exposure  might  help  users  to  better 
understand  the  operator  aids  and  could  lead  to  the  formulation  of  strategies  to  further  increase  the 
aids’  usefulness. 
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One  factor  that  may  have  played  a  significant  role  in  the  perfonnance  results  is  the  nature  of  the 
obstacles  that  were  used.  Two  of  the  obstacles,  Jersey  barriers  and  pedestrians,  were  very  salient 
and  perhaps  did  not  benefit  much  from  the  use  of  operator  aids.  The  tall  grass  obstacle  was  much 
more  ambiguous  to  the  robot  and  provided  a  more  unique  challenge  about  which  participants 
needed  to  make  a  decision.  Future  research  should  focus  on  more  ambiguous  obstacles,  such  as 
the  tall  grass,  which  may  or  may  not  cause  issues  for  an  autonomously  navigating  robot. 

However,  it  should  be  noted  that  although  the  use  of  operator  aids  did  not  increase  performance, 
nor  did  the  inclusion  of  aids  reduce  operator  perfonnance.  Thus,  the  need  for  operators  to  attend 
to  additional  information  provided  by  the  aids  did  not  produce  negative  effects  on  perfonnance. 

5.2  Workload 

Results  of  this  study  yielded  some  interesting  results  about  reported  operator  workload  and  the 
use  of  the  operator  aids.  Significant  results  showed  that  a  combination  of  operator  aids  helped  to 
reduce  self-reported  operator  workload.  Interestingly,  even  with  more  overall  information  to 
attend  to,  operators  were  able  to  reduce  their  perceived  workload  in  regards  to  the  robotic  asset 
under  their  control.  Participants  in  this  study  commented  that  the  long-term  and  short-tenn 
planners  gave  them  a  good  idea  of  where  the  robot  was  trying  to  get  to  and  the  addition  of  the 
obstacle  map  let  them  know  why,  based  on  what  the  robot  was  “seeing.” 

Looking  at  the  individual  workload  measures  revealed  mental  and  temporal  workload  as  key  in 
the  overall  reduction  of  workload.  This  could  be  seen  as  an  indicator  about  participants’  belief 
that  the  operator  aids  made  it  easier  for  them  to  understand  the  robotic  assets  actions  and  intents 
in  a  quick  and  efficient  manner.  Not  surprisingly,  physical  workload  was  not  affected,  given  the 
primarily  cognitive  nature  of  the  task.  Additionally,  the  lack  of  significant  results  for 
perfonnance,  effort,  and  frustration  may  be  due  to  the  relative  simplicity  of  the  task.  In  future 
research,  more  difficult,  complex,  or  taxing  tasks  may  see  different  results  and  are  worth 
studying. 

We  do  not  fully  understand  the  “spike”  in  the  composite  workload  index  for  the  long-tenn 
planner  and  Obstacle  Map  Condition  (Condition  3).  At  this  time,  it  is  unknown  why  the  LTP  and 
Obstacle  Map  condition  was  rated  as  higher  workload  than  LTP  by  itself  (Condition  4)  and  as 
high  as  the  control  condition  (Condition  7). 

Further,  the  overall  workload  results  remained  relatively  low  across  conditions  throughout  the 
study.  This  is  a  sign  that  the  task  itself  may  have  been  not  challenging  enough  to  produce 
significant  workload  challenges  for  the  participants.  If  the  task  was  shown  to  be  too  easy,  this 
also  may  explain  why  the  covariate  measures,  which  have  been  shown  to  be  significant  in 
previous  studies,  did  not  yield  any  significant  results  here.  That  is,  all  participants  could  perform 
the  tasks  well,  regardless  of  their  individual  spatial  ability.  In  future  studies,  task  difficulty 
should  be  advanced  to  levels  that  ensure  an  appropriate  amount  of  workload  so  that  “floor 
effects”  such  as  these  aren’t  a  factor. 
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The  results  from  the  workload  measure  analysis  were  also  supported  by  the  evaluation  of  user 
preferences.  The  preferences  indicated  that  operators  preferred  having  information  coming  from 
multiple  aids  to  help  them  gain  a  complete  understanding  of  the  situation. 

The  results  from  the  3-D  versus  2-D  usage  question  provide  ideas  about  how  operators  would 
spend  their  time  monitoring  the  views  provided  by  a  robotic  asset.  The  majority  of  operators’ 
time  in  this  study  was  spent  on  the  3-D  display;  however,  the  exact  reason  is  still  not  known. 
Future  research  should  investigate  if  this  preference  was  due  to  the  usefulness  or  effectiveness  of 
the  aids  in  one  display  type  versus  another  (e.g.,  3-D  displays  provide  more  meaningful 
information)  or  simply  a  byproduct  of  how  quickly  infonnation  could  be  understood  in  one 
display  type  versus  another  (e.g.,  2-D  displays  conveyed  relevant  infonnation  to  operators  in  a 
more  simple  and  quick  way  to  comprehend). 

5.3  Conclusions 

Overall,  even  with  a  lack  of  significant  perfonnance  outcomes,  other  results  of  this  research  may 
be  viewed  positively  in  that  operator  aids  were  shown  to  create  an  environment  in  which 
operators  can  reduce  cognitive  and  temporal  workload,  although  without  differences  in 
perfonnance.  Again,  more  extensive  training  in  the  use  of  the  operator  aids  and  development  of 
strategies  in  the  use  of  the  aids  might  help  to  realize  greater  improvements  in  workload  reduction 
along  with  potential  perfonnance  increases. 

We  plan  to  conduct  a  similar  study  using  operator  aids  in  a  field  study  environment  with  Soldier 
participants.  This  field  study’s  results  will  be  compared  against  this  cunent  study  to  support  (or 
contradict)  the  current  findings.  Another  area  of  interest  moving  forward  with  this  research  is 
trust.  Operator  aids  may  be  able  to  engender  trust  of  Soldiers  in  robotic  assets.  By  creating  more 
salient  interfaces  for  robotic  assets,  Soldiers  could  have  a  greater  understanding  of  robotic  assets 
and  their  inner  workings.  This  understanding  could  then  help  Soldiers  to  maintain  higher  levels 
of  appropriate  trust,  and  therefore  mission  effectiveness,  within  human-robot  teams  (Kim  and 
Hinds,  2006). 
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Appendix  A.  Examples  of  Ishihara  Color  Vision  Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


21 


Color  Vision  Test 


What  numbers  do  you  see  revealed  in  the  patterns  of  dots  below?  Please  record 
the  number  on  the  answer  sheet  or,  if  you  do  not  see  a  number,  write  “NONE.” 


Please  turn  the  page  to  continue. . . 
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Appendix  B.  Example  of  Guilford-Zimmerman  Spatial 
Visualization  Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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The  Guilford-Zimmerman  Aptitude  Survey 


Part  6/Spatial  Visualization 


Name 


Date _  Score -  Sex:  M  F 


INSTRUCTIONS. 

This  is  a  test  of  how  well  you  are  able  to  visualize  spatial  position.  In  each  item  you  are  to  note  how  the  clock  would 
move  if  it  were  moved  as  indicated  by  the  arrow  on  the  sphere. 

Here  are  some  sample  items. 


The  first  picture  at  the  left  shows  a  clock.  Next  to  it  is  a  sphere  with  an  arrow  marked  on  it.  The  arrow  shows  how  the 
clock  is  to  be  moved.  This  move  is  illustrated  (in  two  steps)  in  the  picture  below.  When  the  clock  is  moved  to  the  one- 
quarter  turn  shown  by  the  arrow,  it  is  then  in  position  B.  B  is  therefore  the  correct  answer.  You  would  record  this  by 
darkening  the  answer  space  right  below  B  on  your  answer  sheet.  (But  do  not  record  answers  to  sample  items.) 


Original 

Position 


Position  after  the  move 
has  been  completed 
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Appendix  C.  Example  of  Guilford-Zimmerman  Spatial 
Orientation  Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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The  Guilford-Zimmerman  Aptitude  Survey 


Part  5/Spatial  Orientation 


Name 


Date -  Score _  Sex:  M  F 


INSTRUCTIONS. 

This  is  a  test  of  your  ability  to  see  changes  in  direction  and  position.  In  each  item  you  are  to  note  how  the 
position  of  the  boat  has  changed  in  the  second  picture  from  the  original  position  in  the  first  picture. 


Here  is  Sample  Item  1. 

These  bars  represent  the  boat’s 
prow. 


This  is  the  correct  answer.  It  shows 
that  the  prow  of  the  boat  has  drop¬ 
ped  below  the  aiming  point.  ■ 


(If  the  prow  had  risen,  instead  of 
dropped,  the  correct  answer  would 
have  been  C,  instead  of  D.) 


These  are  the  five  possible  answers  to  the  item. 


This  is  the  prow  (front  end)  of  a 
motor  boat  in  which  you  are  riding. 

This  is  the  aiming  point.  It  is  the 
exact  spot  you  would  see  on  land 
if  you  sighted  right  over  the  point 
of  the  prow. 


This  is  the  same  aiming  point 
shown  above.  Note  that  the  prow 
has  dropped  below  it. 


To  work  each  item:  First,  look  at  the  top  picture  and  see  where  the  motor  boat  is  headed.  Second,  look  at  the 
bottom  picture  and  note  the  CHANGE  in  the  boat’s  heading.  Third,  mark  the  answer  that  shows  the  same  change  on 
the  separate  answer  sheet. 


Try  Sample  Item  2. 

This  also  shows  that  the  prow  of 
the  boat  is  to  the  right  of  the  aiming 
point.  So,  it  is  the  correct  answer.  — 


(If  the  boat  had  turned  to  the  left, 
instead  of  to  the  right,  the  correct 
answer  would  have  been  A.) 


This  is  the  aiming  point. 


This  is  the  same  aiming  point. 
The  motor  boat  is  now  headed  to 
the  right  of  it. 
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Appendix  D.  Example  of  NASA-TLX  Perceived  Workload  Assessment 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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RATING  SCALE  DEFINITIONS 


TITLE 


MENTAL 

DEMAND 


PHYSICAL 

DEMAND 


TEMPORAL 

DEMAND 


PERFORMANCE 


EFFORT 


FRUSTRATION 

LEVEL 


ENDPOINTS 


DESCRIPTIONS 


LOW/HIGH  How  much  mental  and 

perceptual  activity  was  required 
(e.g.  thinking,  deciding,  calculating, 

remembering,  looking,  searching,  etc.)? 
was  the  task  easy  or  demanding,  simple 
or  complex,  exacting  or  forgiving? 


LOW/HIGH  How  much  physical  activity  was 

required  (e.g.  pushing,  pulling, 
turning,  controlling,  activating, 
etc.)?  Was  the  task  easy  or 

demanding,  slow  or  brisk,  slack 
or  strenuous,  restful  or  laborious? 


LOW/HIGH  How  much  time  pressure  did  you 

feel  due  to  the  rate  or  pace 

at  which  the  task  or  task  elements 
occurred?  was  the  pace  slow  and 
leisurely  or  rapid  and  frantic? 


GOOD/POOR  How  successful  do  you  think  you 

were  in  accomplishing  the  goals 

of  the  task  set  by  the  experimenter 
(or  yourself)?  How  satisfied  were 
you  with  your  performance  in 
accomplishing  these  goals? 


LOW/HIGH  How  hard  did  you  have  to  work 

(mentally  and  physically)  to 
accomplish  your  level  of 
performance? 

LOW/HIGH  How  insecure,  discouraged,  irritated 

stressed,  and  annoyed  versus  secure, 
gratified,  content,  relaxed,  and 
complacent  did  you  feel  during  the  task? 
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Scoring  Form  1 


1.  Mental  Demand  -  Individual 


Low 

2.  Physical  Demand  -  Individual 

1 

1 

1 

1 

1 

High 

Low 

3.  Temporal  Demand  -  Individual 

1 

1 

1 

1 

1 

High 

Low 

4.  Performance  -  Individual 

1 

1 

1 

1 

1 

High 

Good 

5.  Effort  -  Individual 

1 

1 

1 

1 

1 

Bad 

Low 

6.  Frustration  -  Individual 

1 

1 

1 

1 

1 

High 

Low 

High 

29 


Bibliography 


Bames,  M.;  Jentsch,  F.;  Chen,  J.  Y.  C.;  Haas,  H.;  Cosenzo,  K.  Soldier-Robot  Teams:  Six  Years 
of  Research.  Proceedings  of  the  5  4"d  Annual  Meeting  of  the  Human  factors  and  Ergonomics 
Society,  San  Francisco,  CA,  September  2010. 

Sellers,  B.;  Fincannon,  T.;  Jentsch,  F.  The  Effects  of  Autonomy  and  Cognitive  Abilities  on 
Workload  and  Supervisory  Control  of  Unmanned  Systems.  Proceedings  of  the  5  6nd  Annual 
Meeting  of  the  Human  factors  and  Ergonomics  Society,  Boston,  MA,  September  2012, 
1039-1043. 


30 


List  of  Symbols,  Abbreviations,  and  Acronyms 


2-D 

two-dimensional 

3-D 

three-dimensional 

ANOVA 

analysis  of  variance 

ANS 

Autonomous  Navigation  System 

ARL 

U.S.  Anny  Research  Laboratory 

ATO 

Anny  Technology  Objective 

COA 

course  of  action 

COMBO 

combined  short-term  and  long-tenn  planners 

GDRS 

General  Dynamics  Robotics  System 

HRI 

human-robot  interaction 

IMPRINT 

Improved  Performance  Research  Integration  Tool 

LADAR 

laser  radar 

LTP 

long-tenn  planner 

NCO 

Noncommissioned  Officer 

OM 

obstacle  map 

SA 

situation  awareness 

SOURCE 

Safe  Operations  Using  Robotic  in  Complex  Environments 

SPSS 

Statistical  Package  for  the  Social  Sciences 

STP 

short-tenn  planner 

TARDEC 

Tank  Automotive  Research,  Development,  and  Engineering  Center 

TLX 

Task  Load  Index 

UGV 

unmanned  ground  vehicle 

WMI 

Warfighter  Machine  Interface 
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